Papers with hierarchical transformer architecture
Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. |
| Approach: | They propose a hierarchical transformer that quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a transformer architecture. |
| Outcome: | The proposed model outperforms existing speech language models in word error rate, speech quality, and speaker similarity. |
Modelling Temporal Document Sequences for Clinical ICD Coding (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on the ICD coding task focus on extracting codes from the discharge summary, but there is potential to automate the task by identifying relevant information from clinical notes. |
| Approach: | They propose a hierarchical transformer architecture that uses text across the entire sequence of clinical notes in each hospital stay for ICD coding. |
| Outcome: | The proposed model exceeds the state-of-the-art when using only discharge summaries as input and achieves performance improvements when all clinical notes are used as input. |